CamelCamelCamel API Design Decisions

Understand design considerations for the CamelCamelCamel API service.

This lesson will discuss design decisions paramount to creating an effective C3 service. Mainly, we’ll decide on a suitable architectural style, data format, and HTTP version for an effective C3 service.

Design overview#

Let’s start by understanding the architecture and workflow of the C3 service before diving into a discussion of design decisions because C3’s design architecture and workflow will enable us to make suitable decisions. Notice the synchronous communication of the C3 service with Amazon, a distinct service provider. The C3 API uses search, pub-sub, and product services. These services interact with Amazon’s services to provide data to potential customers. The following illustration demonstrates a high-level architecture of the workings of a C3 service:

Working architecture of a C3 service
Working architecture of a C3 service

The following table gives an overview of each of the components involved in a C3 service:

Components and Services Details

Component or Service

Details


API Gateway

  • Client to C3 service
  • C3 to Amazon service
  • Analyzes and routes the requests to appropriate services
  • Authenticates and authorizes requests
  • Limits address rate to throttle requests

C3 Service

Search service

  • Handles client's search requests

Pub-sub service

  • Handles requests to add products to the watch list
  • Notifies users about price drops


Product service

  • Recommends products to users
  • Interacts continuously with Amazon service to update data
  • Publishes an event of the price drop on the pub-sub after data analysis

Amazon Service


Search service

  • Handles search requests made by the C3 service to tackle client's search requests

Simple queuing service (SQS)

  • Stores requests to be processed in a queue

Product advertising service

  • Handles periodic user requests forwarded by SQS

Workflow of C3#

The C3 service works synchronously with the Amazon service. Let’s discuss the workflow of each service according to the functional requirements. Requests to C3’s API gateway can interact with any of the subservices of C3 according to the required functionality, as described below:

  • Search product: If it's a search query from a user, the API gateway sends it to the search service of the C3. The C3’s search service further collaborates with Amazon’s search service. Interacting with Amazon’s search service allows C3 to produce the most accurate results for the client. After receiving a query from C3, Amazon’s search service processes it, generates results, and delivers them back to C3’s search service. It is now C3’s responsibility to process data and send a response back to the client.

  • Adding a product to price watch: If clients want to receive a price drop alert of a product that was searched for earlier using a search service or that was suggested by the C3 service, they need to subscribe to the product with a desired price. Such a request is forwarded to the pub-sub service from the API gateway. It keeps products as topics, and whenever a price drop (event) occurs, it notifies the client (subscriber). To evaluate price drops, the pub-sub service acquires information from the product service of C3. Similarly, clients can get a list of the products from the pub-sub service that were added to their watch list.

Points to Ponder

Question 3

Which type of notification, i.e., email, text, or push notification, is sent to the subscriber from the pub-sub service if a price drop occurs?

Hide Answer

Generally, the email-based notification is sent to the subscriber by the C3 service. However, the pub-sub service also enables other notification types, such as text or push notifications on the address defined by the subscriber in the callBack URL of the subscribe request.

3 of 3

  • Price timeline: The price timeline is acquired using the product service of C3. The product service communicates with Amazon’s product advertising service to get up-to-date information about the products. It uses the SQS service of Amazon to send periodic requests to Amazon’s product advertising service.
    The data received in response is compared with local data of the C3’s product service. The C3’s database gets updated in case of updated results from Amazon’s product service. It also informs the pub-sub service about the prices and details of the products added to the watch list. The workflow of SQS and product advertising API is discussed below:

    • SQS service: The PA-API of Amazon allows a limited number of requests per user since there are otherwise millions of interactions daily. SQS provides queuing functionality to perform tasks (processing requests in our case) periodically. C3 prepares several requests and pushes them to the SQS. The SQS processes each request from the queue. The pattern of each request is that it takes query parameters and asks Amazon’s product advertising service about the data based on the queries. It delivers the response to C3’s product service, as shown in the illustration below.

    • Product advertising service: This service of Amazon provides up-to-date information about millions of products. Requests from SQS are forwarded to the product advertising service, which processes the request, prepares a response, and sends it back to the requesting service.

Note: How periodic requests work is that the PA-API defines rate limiting to address millions of users, so the queue is filled with requests to be sent periodically according to the allowed number of requests in a specified period.

C3's product service communicating with Amazon's product service using SQS
C3's product service communicating with Amazon's product service using SQS

Design considerations#

This section discusses the design considerations of architectural styles, data formats, and HTTP protocols.

Architectural styles#

We have four layers of communication between a client and synchronous services. These layers are listed below:

  • Client to API gateway

  • API gateway to C3 service

  • C3 service to API gateway

  • API gateway to Amazon service

Different operations are performed through the request-response architecture between each layer of communication. All the operations, like searching, reading price history, and sending a subscription request, are simple CRUD operations and resource-oriented. Therefore, we can use the REST API architectural style for communication due to its simple and standardized way of managing resources. Moreover, from the workflow, it is clear that there is no interaction between multiple resources simultaneously for any operation. Though we have synchronous communication between resources, we do not fetch data from multiple resources; therefore, graphQL is not a viable option. Moreover, C3 service depends on Amazon due to the synchronous communication. REST is also preferred due to its compatibility with Amazon.

Architectural style from client to service
Architectural style from client to service

Point to Ponder

Question

As we discussed, customers are notified about price drop alerts by the C3 service. Is it possible to use REST to send notifications?

Hide Answer

REST purely has a request-response architecture. With REST, sending notifications from the server is not possible. The notifications are sent from the server whenever an event of a price drop occurs without a user’s request. So, we can use event-driven protocols for such communication.

Communication protocols#

The choice of a suitable HTTP protocol is a bit more complex in this case. That is because we have to consider both the communication between the client and C3 service and the C3 service and Amazon’s API. The operations are in the simple request-response format and do not need multiple responses (continuous stream of data) for a single request. We can use a persistent connection of HTTP/1.1 to perform simple CRUD operations while interacting with the C3 service. Most applications use the HTTP/1.1 protocol for simple interaction with APIs due to its compatible infrastructure—and so does Amazon. Therefore, HTTP/1.1 seems suitable for the C3 service to have compatible communication with the Amazon service.

The pub-sub service, however, uses event-driven protocols such as WebSub to notify customers. So, for event-driven communication in the C3 service, we’ll use the WebSub protocol.

Although we can have the advantage of low latency by using HTTP/2.0 protocol to fetch a large amount of data, it incurs extra complexity to implement.

Communication protocols from client to service
Communication protocols from client to service

Point to Ponder

Question

Why is WebSub protocol used to send notifications for price drop alerts to clients? Why aren’t other event-driven protocols used?

Hide Answer

We have multiple event-driven protocols such as WebSub, Webhooks, SSE, and WebSockets. WebSocket protocol is suitable for two-way communication, which isn’t required in our case. WebHooks is used for server-server communication, which isn’t our requirement since C3 sends notifications to the clients. SSE and WebSub provide the same features, i.e., sending notifications to the clients, but WebSub takes the lead as it provides an extra security layer (verification of intent.) That’s the most important reason to opt for the WebSub protocol.

Data formats#

The choice of data format is relatively straightforward in this case. Our service depends on Amazon, which uses JSON as its data format. We may consider conversion because data format is an important element in the API design, but JSON also seems like a viable option for our API due to readability and compactness.

Point to Ponder

Question

Suppose the communication between services isn’t dependent on compatibility. How and where can binary data format benefit the C3 service?

Hide Answer

The binary data format is suitable for transferring large data between applications. If compatibility is not an issue, then you can use binary format for the following services:

  • Communication between C3’s search service and Amazon’s search service.
  • Communication between C3’s product service and Amazon’s product service.

This is because the above-mentioned services are server-to-server communication, and you can use binary format for faster data delivery.

Summary#

The design decisions are summarized in the following table:

Design Considerations

Client to API Gateway and C3 Services

C3 to API Gateway and Amazon services

C3 to Subscribers

Architecture Style

REST

REST

Event-Driven

Data Format

JSON

JSON

JSON

Protocols

HTTP/1.1

HTTP/1.1

WebSub

Requirements of the CamelCamelCamel API

API Model for CamelCamelCamel Service